fix(tmux): launch the server with exit-empty off to survive mass teardown - #599
fix(tmux): launch the server with exit-empty off to survive mass teardown#599klabulan wants to merge 1 commit into
Conversation
By default tmux terminates its whole server process the instant the last session closes. During a mass teardown (many sessions ending near-simultaneously), a transient zero-session moment tears the entire server down -- taking every other session's panes with it at once (the whole-server-death incident). Set the server-wide 'exit-empty off' option on session-create (before new_session; server.cmd starts the server if needed), so the server survives an empty moment. Backend belt, HOME-independent (applies to whatever uid runs cao-server); a HOME .tmux.conf set is the ops-side complement. Set on every create_session rather than cached so it survives an externally-killed-and-recreated server; best-effort, so a failure never blocks a launch. Tests: create_session issues set-option -s exit-empty off; a set-option failure does not abort the launch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 6f13d9f)
There was a problem hiding this comment.
Pull request overview
This PR hardens TmuxClient.create_session() against a production race where the tmux server exits during mass teardown due to the default exit-empty on, causing a whole-server shutdown and collateral pane loss.
Changes:
- Add a best-effort helper to set the server-wide tmux option
exit-empty off. - Invoke that helper at the start of
create_session()to keep the server alive across transient zero-session windows. - Add tests verifying the
set-option -s exit-empty offcall and that failures don’t block session creation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/clients/test_tmux_client.py | Adds unit tests for setting exit-empty off and ensuring failures are non-blocking. |
| src/cli_agent_orchestrator/clients/tmux.py | Implements best-effort exit-empty off configuration and calls it from create_session(). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| tmux.create_session("ses", "my-window", "tid1", str(tmp_path)) | ||
|
|
||
| tmux.server.cmd.assert_any_call("set-option", "-s", "exit-empty", "off") |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #599 +/- ##
=======================================
Coverage ? 91.32%
=======================================
Files ? 182
Lines ? 24415
Branches ? 0
=======================================
Hits ? 22297
Misses ? 2118
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
haofeif
left a comment
haofeif
left a comment
There was a problem hiding this comment.
I am requesting changes for one reproduced P2 issue. The mock-based tests pass, but the clean-server path does not establish the option this PR promises: after the first real TmuxClient.create_session() on an isolated socket, tmux still reports exit-empty on; only a second session creation changes it to off. I found no other blocking issues.
| a session launch. | ||
| """ | ||
| try: | ||
| self.server.cmd("set-option", "-s", "exit-empty", "off") |
There was a problem hiding this comment.
[P2] Start the tmux server before setting exit-empty
Server.cmd() does not start a server for set-option. With no tmux server running, this command returns error connecting ... and status 1. libtmux 0.51's Server.cmd() returns that result instead of raising, so this except does not log the failure either. new_session() then starts a fresh server with the default exit-empty on.
I reproduced the exact client path on an isolated socket: after the first create_session(), show-options -s exit-empty returned on; after a second create_session(), it returned off. The first call can be the only session creation in a CAO run because workers are added as windows, and the same gap returns after an external server restart, so that lifecycle is still exposed to the teardown/create race this PR is meant to close. Start and configure the server in one tmux command sequence (for example start-server ; set-option ...), inspect the returned status, and add a real no-server regression test.
By default tmux terminates its whole server process the instant the last session closes (
exit-empty on). During a mass teardown — many sessions ending near-simultaneously — that races CAO creating the next session against the server vanishing: a momentary zero-session window tears the entire server down, taking every other session's panes with it at once (a whole-server-death incident observed in production).Fix:
TmuxClient.create_session()sets the server-wideexit-empty offoption (set-option -s exit-empty off) beforenew_session(server.cmd starts the server if it is not already up), so the server survives a transient empty moment. Best-effort — a failure is logged and never blocks a launch — and set on every create_session so it survives an externally-killed-and-recreated server.Tests: create_session issues
set-option -s exit-empty off; a set-option failure does not abort the launch. Verified live against real tmux 3.4: after create_session,show-options -s exit-emptyreturnsoff.Surfaced in production (harness-control#845).
🤖 Generated with Claude Code